Image processing apparatus, image processing method, and program

ABSTRACT

An image processing apparatus includes a correlating unit configured to acquire correlation information that correlates a first three-dimensional image of a target object with a second three-dimensional image of the target object, and a corresponding cross-sectional image generation unit configured to generate a corresponding cross-sectional image of one of the first three-dimensional image and the second three-dimensional image, if a cross section is set on the other of the first three-dimensional image and the second three-dimensional image, based on the correlation information.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to a medical system capable of processingan image captured with a medical image collection apparatus. Morespecifically, the present invention relates to a technique capable ofassociating a plurality of cross-sectional images with each other andgenerating corresponding cross-sectional images from differentthree-dimensional images.

2. Description of the Related Art

In the medical field physicians diagnose a site of lesion while readinga medical image of a target object displayed on a monitor. Tomographicimages (i.e., three-dimensional images) of the inner state of targetobjects are widely used as medical images.

The medical image collection apparatus (i.e., the modality) capable ofcapturing tomographic images is, for example, an ultrasonic imagediagnosis apparatus (ultrasonic apparatus), an Optical CoherenceTomography (OCT apparatus), a Magnetic Resonance Imaging apparatus (MRIapparatus), an X-ray Computed Tomography apparatus (X-ray CT apparatus),or a combination of like apparatuses.

For example, in the field of mammary gland imaging, the procedure ofimaging diagnosis may include identifying a lesion position in a breastbased on an image captured with an MRI apparatus and observing a stateof the identified lesion with an ultrasonic apparatus.

In this case, an image capturing protocol in the mammary gland imagingdepartment generally includes performing an imaging operation with theMRI apparatus in a state where a diagnosis target is held in a proneposition and then performing an imaging operation with the ultrasonicapparatus in a state where the diagnosis target is held in a supineposition.

In this case, considering a deformation of the breast occurring when theposture of the diagnosis target changes in the above described imagecapturing operations, a physician obtains an estimated position of thesupine position lesion based on the position of the lesion identifiedbased on the prone position MRI image. Then, the physician operates theultrasonic apparatus to capture an image of the lesion at the estimatedposition of the lesion.

However, the deformation of a breast occurring due to a change inimage-capturing posture is very large. Therefore, the position of alesion in the supine position estimated by the physician may greatlydiffer from an actual lesion position. To solve this problem, it may beuseful to use a conventional method for generating a virtual supineposition MRI image that can be obtained by performing deformationprocessing on a prone position MRI image.

The position of a lesion in the virtual supine position MRI image can becalculated based on information relating to the deformation from theprone position to the supine position. Alternatively, the position of alesion on an image can be directly obtained by reading the generatedvirtual supine position MRI image.

If the deformation processing is accurate enough, the actual lesion inthe supine position will be present in the vicinity of the estimatedlesion position on the virtual supine position MRI image.

Further, not only calculating the position of a lesion on the supineposition MRI image that corresponds to the actual lesion position on theprone position MRI image but also displaying corresponding crosssections of the prone position MRI image and the supine position MRIimage may be often required.

For example, the physician may want to precisely observe a state of thelesion on the original image. To this end, it is necessary to display across-sectional image of a non-deformed prone position MRI image thatcorresponds to a cross section including the lesion designated on adeformed virtual supine position MRI image.

In addition, the physician may want to confirm how across section of thenon-deformed prone position MRI image changes to a cross section of thedeformed virtual supine position MRI image.

Currently, there is a conventional method for displaying correspondingcross-sectional images taken from two three-dimensional images that aredifferent from each other in deformation state. Japanese PatentApplication Laid-Open No. 2008-073305 (hereafter “JP 2008-073305”)discloses an example of such a conventional method.

According to the method disclosed in JP 2008-073305, processing to befirst executed is deforming a previous three-dimensional image so as tofit to a present three-dimensional image. Further, processing to be nextexecuted is displaying a present cross-sectional image together with aprevious cross-sectional image in such a manner that two images arearranged symmetrically in the right and left direction or symmetricallyin the up and down direction.

In addition, Japanese Patent Application Laid-Open No. 2009-090120(hereafter “JP 2009-090120”) discloses a method of displaying two imageslices positioned on the same plane. Specifically, in the methoddisclosed in JP 2009-090120, an image slice is designated in one imagedata set to discriminate a corresponding image slice in the other imagedata set, so as to display the two image slices positioned on the sameplane.

However, according to the method discussed in JP 2008-073305, segmentedcross sections corresponding to each other are extracted after theprevious three-dimensional image and the present three-dimensional imageare deformed to have an identical shape. That is, processing time andresources are required to obtain deform the images and obtain theidentical shape.

Therefore, the method discussed in JP 2008-073305 is unable to displaycorresponding cross-sectional images while maintaining differences inshape.

On the other hand, the method discussed in JP 2009-090120 merelysuggests selecting image slices from respective image data sets.Therefore, in ordinary cases, the method discussed in JP 2009-090120 isunable to appropriately generate a corresponding cross-sectional imagein the other image data set so as to correspond to a cross-sectionalimage designated in one image data set.

SUMMARY OF THE INVENTION

The present invention is directed to a technique capable of generatingcorresponding cross-sectional images from different three-dimensionalimages even if differences in shape between the images are maintained.

According to an aspect of the present invention, an image processingapparatus includes a correlating unit configured to acquire correlationinformation that correlates a first three-dimensional image of a targetobject with a second three-dimensional image of the target object, and acorresponding cross-sectional image generation unit configured togenerate a corresponding cross-sectional image of one of the firstthree-dimensional image and the second three-dimensional image, if across section is set on the other of the first three-dimensional imageand the second three-dimensional image, based on the correlationinformation.

Further features and aspects of the present invention will becomeapparent to persons having ordinary skill in the art from the followingdetailed description of exemplary embodiments with reference to theattached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated in and constitute apart of the specification, illustrate exemplary embodiments, features,and aspects of the invention and, together with the description, serveto explain the principles of the invention.

FIG. 1 is a block diagram illustrating a device configuration of animage processing apparatus according to a first exemplary embodiment ofthe present invention.

FIG. 2 is a block diagram illustrating a basic configuration of acomputer that can execute a software program capable of realizingfunctional components of the image processing apparatus according to thefirst exemplary embodiment of the present invention.

FIG. 3 is a flowchart illustrating a procedure of the entire processingaccording to the first exemplary embodiment of the present invention.

FIG. 4 illustrates an example synthesis display method according to thefirst exemplary embodiment of the present invention.

FIG. 5 is a block diagram illustrating a device configuration of animage processing apparatus according to a second exemplary embodiment ofthe present invention.

FIG. 6 is a flowchart illustrating an example procedure of the entireprocessing according to the second exemplary embodiment of the presentinvention.

FIG. 7 illustrates an example synthesis display mode according to thesecond exemplary embodiment of the present invention.

DESCRIPTION OF THE EMBODIMENTS

Various exemplary embodiments, features, and aspects of the inventionwill be described in detail below with reference to the drawings.

An image processing apparatus according to the present exemplaryembodiment calculates an approximate plane, i.e., a plane approximatinga corresponding cross section (curved surface) of an MRI image under afirst deformation condition, which corresponds to a cross section in theMRI image under a second deformation condition. Further, the imageprocessing apparatus displays an image of the cross section togetherwith an image of the corresponding cross section.

In the present exemplary embodiment, the case where a breast of a humanbody is an example of a target object will be described. Further, in thepresent exemplary embodiment, it is assumed that the first deformationcondition represents a state where the breast is held in a proneposition with respect to a gravitational direction and the seconddeformation condition represents a state where the breast is held in asupine position with respect to the gravitational direction.

In the present exemplary embodiment, it is assumed that a supineposition MRI image is a virtual image that can be obtained by performingdeformation processing on a prone position MRI image having beenactually obtained. Further, it is assumed that a cross sectionrepresenting a tomographic image that can be obtained by momentarilycapturing an image of the target object held in the supine position withan ultrasonic apparatus is set as a cross section of the supine positionMRI image. Hereinafter, the image processing apparatus according to thepresent exemplary embodiment is described below.

FIG. 1 is a block diagram illustrating an example configuration of animage processing apparatus according to the present exemplaryembodiment. As illustrated in FIG. 1, an image processing apparatus 100according to the present exemplary embodiment is connected to a firstimage capturing apparatus 180, a shape measurement apparatus 184, and adata server 190.

The first image capturing apparatus 180 is, for example, an ultrasonicapparatus that is configured to capture an image of each target objectheld in the supine position with a probe (not shown). It is assumed thatduring the image-capture operation the probe transmits an ultrasonicsignal towards the target object and receives the ultrasonic signalreturning from target object in a known manner.

It is further assumed that the position and orientation of the probeduring an image-capturing operation can be measured by a position andorientation sensor, also in a known manner. Further, it is assumed thatthe measured position and orientation of the probe can be converted intoposition and orientation information representing an ultrasonictomographic image in a reference coordinate system (i.e., a coordinatesystem being set in a space including the target object as a referenceobject), and is associated with the ultrasonic tomographic image.Accordingly, it is assumed that position and orientation information isattached to each input ultrasonic tomographic image.

The position and orientation sensor can be any type of sensor that isconfigured to obtain measurement information representing the positionand orientation of the probe. And a description is omitted since it isperipheral technology. Ultrasonic tomographic images captured by thefirst image capturing apparatus 180 are defined in the referencecoordinate system, and are successively input into the image processingapparatus 100 via a tomographic image acquisition unit 102.

The data server 190 stores three-dimensional images (i.e., firstthree-dimensional images) obtained by capturing images of a targetobject in the prone position with an MRI apparatus serving as a secondimage capturing apparatus 182. The three-dimensional images stored indata server 190 are also converted into images defined in the referencecoordinate system. The first three-dimensional images stored in the dataserver 190 can be input into the image processing apparatus 100 via athree-dimensional image acquisition unit 104.

The shape measurement apparatus 184 is, for example, a range sensor thatcan momentarily measure a surficial shape of a dynamically changingtarget object held in the supine position to obtain shape data. Theshape data obtained by shape measurement apparatus 184 can besuccessively input into the image processing apparatus 100 via a shapedata acquisition unit 105. The shape measurement apparatus 184 is notlimited to the range sensor and can be constituted by any othermeasuring device capable of momentarily measuring the shape of thetarget object (e.g., a stereo image measurement device).

The image processing apparatus 100 includes a plurality of constituentcomponents described below.

The tomographic image acquisition unit 102 successively acquiresultrasonic tomographic images input to the image processing apparatus100 and outputs the acquired ultrasonic tomographic images to a crosssection setting unit 108 and further to an image synthesis unit 120. Inthe present exemplary embodiment, it is assumed that each ultrasonictomographic image is accompanied with position and orientationinformation defined in the reference coordinate system.

The three-dimensional image acquisition unit 104 acquires each firstthree-dimensional image input to the image processing apparatus 100 andfurther outputs the acquired first three-dimensional image to adeformation data generation unit 106 and also to a correspondingcross-sectional image generation unit 111.

The shape data acquisition unit 105 successively acquires shape datainput into the image processing apparatus 100 and outputs the acquiredshape data to the deformation data generation unit 106.

The deformation data generation unit 106 can perform deformationprocessing on the input first three-dimensional image to generate asecond three-dimensional image having a deformed surficial shapesubstantially similar to that of the measured shape data, and can outputthe generated second three-dimensional image to a cross-sectional imagegeneration unit 110.

A correlating unit 107 can obtain “correlation information”, whichrepresents a correspondence relationship between respective coordinatesin the deformation processing from the first three-dimensional image tothe second three-dimensional image. The obtained correlation informationcan be stored in a predetermined storage unit of the image processingapparatus 100 or elsewhere.

The deformation data generation unit 106 can refer to the correlationinformation in associating respective coordinates of the firstthree-dimensional image with respective coordinates of the secondthree-dimensional image.

Further, the deformation data generation unit 106 can calculate athree-dimensional displacement vector group, which is usable toinversely transform the second three-dimensional image back into thefirst three-dimensional image, based on the correlation information andcan output the calculated three-dimensional displacement vector group tothe corresponding cross-sectional image generation unit 111.

The cross section setting unit 108 sets a cross section in the secondthree-dimensional image based on the position and orientationinformation attached to each input ultrasonic tomographic image. To thisend, the cross section setting unit 108 acquires an ultrasonictomographic image from the tomographic image acquisition unit 102.

Then, the cross section setting unit 108 sends the position andorientation (i.e., the attached information) of the ultrasonictomographic image defined in the reference coordinate system to thecross-sectional image generation unit 110 and also to the correspondingcross-sectional image generation unit 111. The position and orientationof the ultrasonic tomographic image serves as position and orientationinformation of a plane representing the cross section in the secondthree-dimensional image.

The cross-sectional image generation unit 110 generates an image of thecross section set in the second three-dimensional image. To this end,the cross-sectional image generation unit 110 acquires the secondthree-dimensional image from the deformation data generation unit 106and the position and orientation of the cross section from the crosssection setting unit 108.

Then, the cross-sectional image generation unit 110 generates across-sectional image based on the acquired second three-dimensionalimage and the acquired position/orientation information, and outputs thegenerated cross-sectional image to the image synthesis unit 120.

The corresponding cross-sectional image generation unit 111 generates acorresponding cross-sectional image in the first three-dimensionalimage, which corresponds to the cross section set in the secondthree-dimensional image, using various information acquired.

Then, the corresponding cross-sectional image generation unit 111outputs the generated image to the image synthesis unit 120.

An example processing that can be implemented by the correspondingcross-sectional image generation unit 111 is described below in moredetail with reference to a flowchart illustrating an example procedureof the entire processing that can be implemented by the image processingapparatus 100.

The image synthesis unit 120 combines an ultrasonic tomographic image, across-sectional image (based on the second three-dimensional image), anda corresponding cross-sectional image (based on the firstthree-dimensional image) and displays a synthesized image. To this end,the image synthesis unit 120 acquires an ultrasonic tomographic imagefrom the tomographic image acquisition unit 102, a cross-sectional imagefrom the cross-sectional image generation unit 110, and a correspondingcross-sectional image from the corresponding cross-sectional imagegeneration unit 111. Then, the image synthesis unit 120 displays thesynthesized image in an appropriate display unit to be described later.

At least a part of the components constituting the image processingapparatus 100 illustrated in FIG. 1 can be realized as an independentstand-alone apparatus. Alternatively, a single computer or a pluralityof computers can be provided with a computer program (a softwareprogram) capable of realizing the above-described functions of the imageprocessing apparatus 100.

A central processing unit (CPU) incorporated in each computer canexecute the installed program to realize the above-described functions.In the present exemplary embodiment, as illustrated in FIG. 1, eachcomponent of the image processing 100 can be constituted by a separatehardware device (e.g., a processing board or CPU). Alternately, eachcomponent of image processing apparatus 100 can be realized byspecifically designed software that is installed on a single computer.

FIG. 2 illustrates a basic configuration of a computer that isconfigured to operate hardware components or software modules capable ofrealizing the functions of the constituent components of imageprocessing apparatus 100 illustrated in FIG. 1.

In FIG. 2, a central processing unit (CPU) 1001 can control an entirecomputer and components attached thereto based on programs and datawhich are stored in a random access memory (RAM) 1002 and a read onlymemory (ROM) 1003. Further, the CPU 1001 can control execution of thesoftware in each unit and can realize the functions of respective units.

The RAM 1002 includes a storage area that can temporarily store programsand data loaded from an external storage device 1007 and a storagemedium drive 1008. The RAM 1002 further includes a work area that can beused by the CPU 1001 when the CPU 1001 performs various processing.

In general, the ROM 1003 stores computer programs and setting data. Akeyboard 1004 and a mouse 1005 are input devices that enable an operatorto input various instructions and target information into the CPU 1001.

A display unit 1006, which is constituted by a cathode ray tube (CRT), aliquid crystal display (LCD) or the like, displays an ultrasonictomographic image, a cross-sectional image, a correspondingcross-sectional image, or a combination thereof. Further, the displayunit 1006 can display a message and a graphical user interface (GUI)screen to aid the operator in accomplishing the various tasks of theimage processing apparatus 100.

The external storage device 1007 is, for example, a hard disk drive orany other massive information storage device capable of storing anoperating system (OS) and the computer programs to be executed by theCPU 1001. Further, in the present exemplary embodiment, any informationdescribed as being already known is stored in the external storagedevice 1007 and can be loaded into the RAM 1002 if it is necessary.

The storage medium drive 1008 reads a stored program or data from astorage medium (e.g., a CD-ROM or a DVD-ROM) according to an instructionsupplied from the CPU 1001 and outputs the read program or data onto theRAM 1002 or to the external storage device 1007.

An interface (I/F) 1009 can be constituted by an appropriate port, suchas an analog video port, a digital input/output port (e.g., IEEE1394),or an Ethernet® port usable to output various information to an externaldevice. The I/F 1009 sends input data to the RAM 1002.

At least part of the above-described functions of the tomographic imageacquisition unit 102, the three-dimensional image acquisition unit 104,and the shape data acquisition unit 105 can be realized by (or attachedto) the I/F 1009.

The above-described functional components 1001 to 1009 are mutuallyconnected via a bus 1010.

FIG. 3 is a flowchart illustrating an example procedure of theprocessing that can be implemented by the image processing apparatus100. In the present exemplary embodiment, the CPU 1001 executes acomputer-executable program that can implement or control the functionsof respective constituent components so as to realize the flowchartillustrated in FIG. 3.

In the present exemplary embodiment, it is assumed that the programcodes relating to the flowchart illustrated in FIG. 3 are alreadyloaded, for example, from the external storage device 1007 to the RAM1002, before starting the following processing.

In step S3000, the image processing apparatus 100 executes, as exampleprocessing that can be implemented by the three-dimensional imageacquisition unit 104, processing for acquiring a first three-dimensionalimage input to the image processing apparatus 100 from the data server190 or second image capturing apparatus 182.

In step S3005, the image processing apparatus 100 executes, as exampleprocessing that can be implemented by the shape data acquisition unit105, processing for acquiring latest shape data momentarily input to theimage processing apparatus 100 from the shape measurement apparatus 184.

In step S3010, the image processing apparatus 100 executes, as exampleprocessing that can be implemented by the deformation data generationunit 106, for generating a second three-dimensional image by applyingdeformation processing on the first three-dimensional image based on thefirst three-dimensional image and the shape data acquired in theabove-described steps S300 and S305.

Further, the image processing apparatus 100 calculates athree-dimensional displacement vector group, which is usable toinversely transform the second three-dimensional image into the firstthree-dimensional image.

More specifically, the correlating unit 107 obtains correlationinformation representing a correspondence relationship betweenrespective coordinates of different images, as information usable togenerate the second three-dimensional image from the firstthree-dimensional image, on condition that the measured shape datasubstantially coincides with the surficial shape of the secondthree-dimensional image.

A method employable to obtain the above-described correlationinformation is, for example, discussed in Y. Hu, D. Morgan, H. U. Ahmed,D. Pendse, M. Sahu, C. Allen, M. Emberton and D. Hawkes, “A statisticalmotion model based on biomechanical simulations,” Proc. MICCAI 2008,Part I, LNCS 5241, pp. 737-744, 2008.

The image processing apparatus 100 performs deformation processing onthe first three-dimensional image referring to the obtained correlationinformation to generate the second three-dimensional image. Further,based on the above-described correlation information, the imageprocessing apparatus 100 calculates a three-dimensional displacementvector group, which is usable to inversely transform voxel positionsconstituting the second three-dimensional image into voxel positionsconstituting the first three-dimensional image in a non-deformed state.

In step S3020, the image processing apparatus 100 executes, as exampleprocessing that can be implemented by the tomographic image acquisitionunit 102, processing for acquiring a latest ultrasonic tomographic imageinput into the image processing apparatus 100 together with attachedinformation representing the position and orientation of the acquiredultrasonic tomographic image defined in the reference coordinate system.

In step S3030, the image processing apparatus 100 executes, as exampleprocessing that can be implemented by the cross section setting unit108, processing for setting the position and orientation informationacquired in step S3020 that represents the ultrasonic tomographic imagedefined in the reference coordinate system, as position and orientationof a plane representing the cross section.

In step S3040, the image processing apparatus 100 executes, as exampleprocessing that can be implemented by the cross-sectional imagegeneration unit 110, for generating a cross-sectional image as an imageobtained by segmenting a predetermined range of the cross section havingbeen set in step S3030 from the second three-dimensional image generatedin step S3010.

In the present exemplary embodiment, a conventionally known method forsegmenting and generating a designated cross-sectional image from athree-dimensional image can be used, although it is not described indetail, it is assumed that persons of ordinary skill on the art ofmedical imaging are familiar with such methods.

Next, the image processing apparatus 100 executes, as example processingthat can be implemented by an approximate plane calculation unit (notillustrated) of the corresponding cross-sectional image generation unit111, processing for calculating an approximate plane that is a planeapproximating a corresponding cross section of the firstthree-dimensional image through the following processing of step S3050to step S3070.

In step S3050, the image processing apparatus 100 executes, as exampleprocessing that can be implemented by the corresponding cross-sectionalimage generation unit 111, processing for segmenting the predeterminedrange of the plane representing the cross section having been set instep S3030 into a lattice pattern divided at equal intervals and settinga group of grid points as a grid point group.

The above-described grid points include, at least, an origin of a crosssection coordinate system that represents a central position of thecross-sectional image. The cross section coordinate system defines thecross section as an XY plane and defines an axis perpendicular to the XYplane as the Z axis.

Further, the above-described grid points include four vertex points(−X_(min2), −Y_(min2)), (X_(min2), Y_(min2)), (−X_(min2), Y_(min2)), and(X_(min2), Y_(min2)) of a rectangular area representing thepredetermined range having been set in step S3040 to segment the imagefrom the cross section.

Further, the above-described grid points include respective endpoints(−X_(min2), 0), (X_(min2), 0) (0, −Y_(min2)), and (0, Y_(min2)) of the Xand Y axes constituting the cross section coordinate system on thecross-sectional image.

Then, the image processing apparatus 100 calculates the positions ofrespective grid points defined in the reference coordinate system usingthe following formula.x _(sn) =x _(in) ·T _(is)

In the above-described formula, x_(in) (=[x_(in) y_(in) z_(in) 1]^(T))is a homogenous coordinate expression in a three-dimensional spacerepresenting the position of an n-th grid point (n=1 to N; N is thetotal number of the grid points) defined in the cross section coordinatesystem.

Further, x_(sn) (=[x_(sn) y_(sn) z_(sn)1]^(T)) represents the positionof the n-th grid point in the reference coordinate system. Further,T_(is) is a 4×4 transformation matrix in the conversion from the crosssection coordinate system to the reference coordinate system, whichrepresents the position and orientation of the cross section having beenset in step S3030. As respective grid points are present on the crosssection, the element z_(in) becomes 0, i.e., z_(in)=0 (n=1 to N).

In step S3060, the image processing apparatus 100 executes, as exampleprocessing that can be implemented by the corresponding cross-sectionalimage generation unit 111, processing for displacing the positions ofrespective grid points calculated in step S3050 based on thethree-dimensional displacement vector group calculated in step S3010.

Then, the image processing apparatus 100 calculates the positions of apoint group (i.e., a corresponding point group) in the firstthree-dimensional image, which correspond to the displaced positions ofthe grid point group in the second three-dimensional image.

More specifically, the image processing apparatus 100 selects, forexample, a voxel constituting the second three-dimensional image that ispositioned most closely to the position x_(sn) of each grid point (n=1to N). Then, the image processing apparatus 100 calculates a positionx_(dn) (n=1 to N) of a corresponding point in the firstthree-dimensional image by adding a three-dimensional displacementvector at the selected voxel position to the position of each gridpoint.

In general, the first three-dimensional image and the secondthree-dimensional image are different from each other in deformationstate. Therefore, the corresponding point group x_(dn) (n=1 to N) cannotbe positioned on the same plane.

In step S3070, the image processing apparatus 100 executes, as exampleprocessing that can be implemented by the corresponding cross-sectionalimage generation unit 111, processing for calculating a plane thatapproximates the corresponding point group based on the positions of thepoint group (i.e., the corresponding point group) of the firstthree-dimensional image calculated in step S3060.

More specifically, the image processing apparatus 100 calculates anapproximate plane that is optimized for the corresponding point groupx_(dn) (n=1 to N) using a general plane fitting method, such as leastsquares method or maximum likelihood estimation method.

Further, if an operator designates a desired approximate planecalculation method using an UI (not illustrated), the correspondingcross-sectional image generation unit 111 calculates an approximateplane using the designated calculation method.

For example, the corresponding cross-sectional image generation unit 111can employ a calculation method including extracting the intensity of animage feature quantity (e.g., an edge) in the vicinity of each gridpoint from the cross-sectional image obtained in step S3040 and thenreferring to the extracted intensity value as a weighting factor for acorresponding point in obtaining an approximate plane using the leastsquares method.

Further, the corresponding cross-sectional image generation unit 111 canemploy a calculation method including defining the position (i.e.,fixing the position) of a point corresponding to the grid pointrepresenting the origin of the cross section as an origin of anapproximate plane coordinate system and then calculating an orientationof an approximate plane optimized for other corresponding points.

In this case, some degrees of freedom may remain with respect toin-plane position and in-plane rotation of the approximate planecalculated in the above-described processing. Therefore, the imageprocessing apparatus 100 further estimates an in-plane moving componentof the approximate plane.

First, the image processing apparatus 100 obtains a crossing point of aperpendicular extending from the point corresponding to the grid pointrepresenting the origin of the cross section and crossing theabove-described approximate plane, and defines the obtained point as theorigin of the approximate plane coordinate system (more specifically,determines an in-plane position of the approximate plane).

Next, the image processing apparatus 100 successively obtains a crossingpoint of a perpendicular extending from each point corresponding torespective grid points positioned on the X axis of the cross section andcrossing the above-described approximate plane, and defines anapproximate straight line optimized for the obtained point group as an Xaxis of the approximate plane coordinate system (more specifically,determines the in-plane rotation of the approximate plane coordinatesystem).

Further, the image processing apparatus 100 calculates vertexcoordinates (−X_(min1), −Y_(min1)), (X_(min1), −Y_(min1)), (−X_(min1),Y_(min1)), and (X_(min1), Y_(min1)) defining a rectangle including allof crossing points of perpendiculars extending from the correspondingpoint group x_(dn) and crossing the above-described approximate plane.

Then, the image processing apparatus 100 defines the calculatedrectangle as the above-described predetermined range usable in the nextstep in which segmenting of the corresponding cross-sectional image isperformed.

In step S3080, the image processing apparatus 100 executes, as exampleprocessing that can be implemented by the corresponding cross-sectionalimage generation unit 111, processing for generating a correspondingcross-sectional image by segmenting the predetermined range of theapproximate plane calculated in step S3070 from the firstthree-dimensional image.

In the present exemplary embodiment, a conventionally known method forsegmenting and generating a designated plane image from athree-dimensional image can be used, although it is not described indetail.

Further, if an operator designates a desired correspondingcross-sectional image generation method using the UI (not illustrated),the corresponding cross-sectional image generation unit 111 generates acorresponding cross-sectional image according to the designatedgeneration method.

For example, the corresponding cross-sectional image generation unit 111can employ a generation method including calculating a curved surfacebased on the positions of the point group (i.e., the corresponding pointgroup) in the first three-dimensional image and then defining pixelvalues of crossing points of perpendiculars extending from the positionsof respective pixels on the approximate plane and crossing the curvedsurface as pixel values of respective pixels of the correspondingcross-sectional image.

According to the above-described generation method, an image of thecorresponding point group projected on the approximate plane can begenerated. In this case, each pixel on a genuine corresponding crosssection (i.e., a curved surface) of the cross-sectional image can bedisplayed in such a manner that the comparison between the generatedimage and the cross-sectional image becomes easy.

In the present exemplary embodiment, instead of calculating theabove-described curved surface, it is useful to calculate a group oftriangular patches having vertex points defined by the correspondingpoint group.

In step S3090, the image processing apparatus 100 executes, as exampleprocessing that can be implemented by the image synthesis unit 120,processing described below. More specifically, the image processingapparatus 100 displays the ultrasonic tomographic image acquired in stepS3020, the cross-sectional image generated in step S3040, and thecorresponding cross-sectional image generated in step S3080 on thedisplay unit 1006.

For example, as illustrated in FIG. 4, the image processing apparatus100 can display a supine position cross-sectional image 400 (i.e., across-sectional image) in such a manner that a nipple 401 substantiallyfaces upward (namely, the chest wall is located at a relatively lowerposition). Further, the image processing apparatus 100 can display aprone position cross-sectional image 402 (i.e., a correspondingcross-sectional image) in such a manner that a nipple 403 substantiallyfaces downward (i.e., the chest wall is located at a relatively upperposition).

In this case, the image processing apparatus 100 determines a relativepositional relationship between two images 400 and 402 in such a mannerthat the supine position cross-sectional image 400 is positioned on theupper side of the screen and the prone position cross-sectional image402 is positioned on the lower side of the screen.

Further, the image processing apparatus 100 determines the layoutrealizing a mirror image relationship between two images 400 and 402,according to which the images 400 and 402 are substantially symmetricalin the vertical direction. To this end, the image processing apparatus100 reverses the prone position corresponding cross-sectional image inthe right and left direction. Further, as illustrated in FIG. 4, anultrasonic tomographic image 404 can be displayed on the right side (orthe left side) of the cross-sectional image 400.

Further, if an operator designates a desired cross-sectional imagedisplay mode using the UI (not illustrated), the image synthesis unit120 displays a cross-sectional image according to the designated displaymode.

For example, the image processing apparatus 100 can employ a method fordisplaying a nipple facing upward (namely, displaying the chest wall ata relatively lower position) in each of the supine positioncross-sectional image 400 and the prone position cross-sectional image402 without reversing the image in the right and left direction.

In other words, with respect to the display mode for the prone positioncross-sectional image 402, an operator can arbitrarily determine whetherto display a nipple facing downward in a state reversed in the right andleft direction or display a nipple facing upward in a state not reversedin the right and left direction.

In step S3100, the image processing apparatus 100 determines whether theentire processing has been completed. For example, an operator can clickan end button disposed on the display unit 1006 with the mouse 1005 toinput a termination instruction to the image processing apparatus 100.If it is determined that the entire processing has been completed (YESin step S3100), the image processing apparatus 100 terminates theprocessing of the flowchart illustrated in FIG. 3.

On the other hand, if it is determined that the entire processing hasnot been completed (NO in step S3100), the processing returns to stepS3005 to repeat the above-described processing in step S3005 to stepS3090 on a newly acquired ultrasonic tomographic image and shape data.

As described above, the image processing apparatus 100 can executeprocessing in a manner capable of realizing the present invention.

As described above, the image processing apparatus according to thepresent exemplary embodiment calculates a plane that approximates acorresponding cross section of a non-deformed three-dimensional imagecorresponding to a cross section of a deformed three-dimensional image.Therefore, the image processing apparatus according to the presentexemplary embodiment can calculate a corresponding cross section withoutdeforming two three-dimensional images into the same shape, and candisplay these images.

Further, the image processing apparatus according to the presentexemplary embodiment can realize a correlated display enabling users toeasily compare a non-deformed cross-sectional image with a deformedcross-sectional image when a deformation occurs due to a change in theposture of a diagnosis target.

In the present exemplary embodiment, the image processing apparatus 100virtually generates a supine position MRI image by deforming a proneposition MRI image. However, the present invention is not limited to theabove-described embodiment. For example, the image processing apparatus100 can capture a supine position MRI image beforehand and can virtuallygenerate a prone position MRI image by deforming the supine position MRIimage.

In this case, the image processing apparatus 100 is required to store athree-dimensional displacement vector group usable in the conversionfrom the non-deformed supine position MRI image into the deformed proneposition three-dimensional image. Further, the present invention can bepreferably employed in a case where a plurality of MRI images under aplurality of deformation conditions is generated by deforming an MRIimage under the first deformation condition. Further, the target objectis not limited to a breast of a human body and can be any otherarbitrary target object.

In the above-described exemplary embodiment, the shape of the targetobject under the second deformation condition is momentarily measuredand then the second three-dimensional image and the three-dimensionaldisplacement vector group are successively calculated. However, if theshape of the target object does not dynamically change under the seconddeformation condition (or if such a change is negligible), it may beuseful to acquire a second three-dimensional image generated beforehand.

In this case, a three-dimensional image (the second three-dimensionalimage) obtained beforehand by deforming the first three-dimensionalimage so that the shape of the target object in the firstthree-dimensional image substantially coincides with the shape of thetarget object in the supine position is stored in the data server 190.

Further, the three-dimensional displacement vector group usable ininversely transforming the deformed second three-dimensional image intothe non-deformed first three-dimensional image is calculated beforehand.The calculated three-dimensional displacement vector group is convertedinto a vector defined in the reference coordinate system and is storedin the data server 190.

In the present exemplary embodiment, the second three-dimensional imagecan be acquired beforehand, for example, using the method discussed inY. Hu, D. Morgan, H. U. Ahmed, D. Pendse, M. Sahu, C. Allen, M. Embertonand D. Hawkes, “A statistical motion model based on biomechanicalsimulations,” Proc. MICCAI 2008, Part I, LNCS 5241, pp. 737-744, 2008.

The shape of the target object in the supine position can be acquired,for example, by measuring a surficial shape of the target object withthe range sensor serving as the shape measurement apparatus 184, astereo image measurement device, or a contact-type digitizer under thesecond deformation condition.

The first and second three-dimensional images and the three-dimensionaldisplacement vector group stored in the data server 190 can be input tothe image processing apparatus 100 via the three-dimensional imageacquisition unit 104.

The three-dimensional image acquisition unit 104 acquires the first andsecond three-dimensional images and the three-dimensional displacementvector group having been input to the image processing apparatus 100.Then, the three-dimensional image acquisition unit 104 outputs theacquired first three-dimensional image and the three-dimensionaldisplacement vector group to the corresponding cross-sectional imagegeneration unit 111. Further, the three-dimensional image acquisitionunit 104 outputs the second three-dimensional image to thecross-sectional image generation unit 110.

According to the above-described method, in a case where the shape ofthe target object does not dynamically change under the seconddeformation condition (or if such a change is negligible), it is easy tocalculate corresponding cross sections of non-deformed and deformedthree-dimensional images and display the calculated images.

In the above-described present exemplary embodiment, the position andorientation of an ultrasonic tomographic image of the target object heldin the supine position momentarily captured by the ultrasonic apparatusis set as position and orientation of a cross section. However, themethod for designating a cross section is not limited to theabove-described method. Therefore, any other method capable of setting across section of a three-dimensional image is usable.

For example, a general method enabling an operator to designate amovement of a cross section in the rotational direction, in therotational angle, and in each axis direction with the mouse 1005 and thekeyboard 1004 is usable. An operator may move and rotate a position andorientation instruction device equipped with a position and orientationsensor, a measured position and orientation value obtained by the sensorcan be acquired as the position and orientation of the cross section.

The second image capturing apparatus 182 according to the presentinvention is not limited to the MRI apparatus described in theabove-described exemplary embodiment. For example, an X-ray CTapparatus, a photo acoustic tomography apparatus, an OCT apparatus, aPositron Emission Tomography (PET), a Single Photon Emission ComputedTomography (SPECT), or a three-dimensional ultrasonic apparatus can beused.

In the above-described exemplary embodiment, the image processingapparatus 100 generates the cross-sectional images from the first andsecond three-dimensional images based on a designated or calculatedcross section in the processing of step S3040 and step S3080.

However, the cross-sectional image to be generated by the imageprocessing apparatus 100 may not be an image formed based on voxelvalues on the cross section even when the image can be generated from athree-dimensional image based on a designated or calculated crosssection.

For example, it may be useful to set a maximum intensity projectionimage as a cross-sectional image in a case where a predetermined rangeis settable in a normal direction about a cross section and a maximumvalue of the voxel value in the normal direction within the set range isobtainable for each point on the cross section.

In the present invention, any image generated with respect to adesignated or calculated cross section as described above can beincluded in the “cross-sectional image” in its broader sense.

In the first exemplary embodiment, the image processing apparatus 100calculates the approximate plane optimized for the corresponding pointgroup using a general plane fitting method, such as the least squaresmethod or the maximum likelihood estimation method. However, theapproximate plane calculation method is not limited to theabove-described method. Therefore, any other method can be employed.

Further, in the first exemplary embodiment, the image processingapparatus 100 designates a cross section in the deformedthree-dimensional image and generates a corresponding cross-sectionalimage that corresponds to the designated cross section in thenon-deformed three-dimensional image.

However, the method for displaying corresponding cross sections of thenon-deformed three-dimensional image and the deformed three-dimensionalimage is not limited to the above-described method.

An image processing apparatus according to a second exemplary embodimentcalculates an approximate plane of a corresponding cross section (i.e.,a curved surface) in a deformed MRI image that corresponds to a crosssection in a non-deformed MRI image based on a concerned position (i.e.,a lesion position) in the MRI image.

Then, the image processing apparatus according to the second exemplaryembodiment generates and displays a cross-sectional image together witha corresponding cross-sectional image.

Hereinafter, only a portion of the image processing apparatus accordingto the second exemplary embodiment, which is different from the imageprocessing apparatus described in the first exemplary embodiment, isdescribed below.

FIG. 5 is a block diagram illustrating an example configuration of theimage processing apparatus according to the second exemplary embodiment.A component similar to that illustrated in FIG. 1 is denoted using thesame reference numeral and the description thereof is not repeated. Asillustrated in FIG. 5, an image processing apparatus 500 according tothe second exemplary embodiment is connected to a data server 590.

The data server 590 stores three-dimensional images (i.e., firstthree-dimensional images) obtained by capturing images of a targetobject in the prone position with the MRI apparatus serving as thesecond image capturing apparatus 182 together with information relatingto a first lesion position indicating the position of a lesion in thefirst three-dimensional image.

Further, the data server 590 stores three-dimensional images (i.e.,second three-dimensional images) obtained by deforming the firstthree-dimensional images beforehand so that a first shape representingthe shape of the target object in the first three-dimensional imagesubstantially coincides with the shape of the target object in thesupine position.

Further, the data server 590 stores a three-dimensional displacementvector group usable in conversion from a non-deformed firstthree-dimensional image to a deformed second three-dimensional image,calculated beforehand as correlation information representing acorrespondence relationship between respective coordinates of differentimages, which is converted into a vector defined in the referencecoordinate system.

The first and second three-dimensional images and the three-dimensionaldisplacement vector group stored in the data server 590 can be input tothe image processing apparatus 500 via a three-dimensional imageacquisition unit 504. Further, the first lesion position data stored inthe data server 590 can be input to the image processing apparatus 500via a position acquisition unit 503.

The position acquisition unit 503 acquires the first lesion positiondata having been input in the image processing apparatus 500 as theposition of a concerned point, and outputs the acquired positionalinformation of the concerned point to a deformation data generation unit506 and a corresponding cross-sectional image generation unit 511.

The three-dimensional image acquisition unit 504 acquires the first andsecond three-dimensional images and the three-dimensional displacementvector group having been input to the image processing apparatus 500.Then, the three-dimensional image acquisition unit 504 outputs theacquired first three-dimensional image to a cross-sectional imagegeneration unit 510. Further, the three-dimensional image acquisitionunit 504 outputs the acquired second three-dimensional image and thethree-dimensional displacement vector group to the correspondingcross-sectional image generation unit 511.

Further, the three-dimensional image acquisition unit 504 outputs thethree-dimensional displacement vector group to the deformation datageneration unit 506. As described above, in the present exemplaryembodiment, the three-dimensional image acquisition unit 504 isfunctionally operable as the above-described correlating unit 107 thatcan acquire correlation information between different images.

The deformation data generation unit 506 calculates a second lesionposition indicating the position corresponding to the first lesionposition in the second three-dimensional image, based on the acquiredfirst lesion position data and the three-dimensional displacement vectorgroup. Further, the deformation data generation unit 506 outputs thecalculated second lesion position to the corresponding cross-sectionalimage generation unit 511.

Alternatively, it is useful to store a second lesion position calculatedbeforehand in the data server 590, instead of causing the deformationdata generation unit 506 to calculate the second lesion position. Inthis case, the position acquisition unit 503 acquires the second lesionposition from the data server 590 and outputs the acquired second lesionposition to the corresponding cross-sectional image generation unit 511.

Across section setting unit 508 acquires information relating to amovement of a plane in the rotational direction, in the rotationalangle, and in each axis direction, which is input to the imageprocessing apparatus 500 by an operator with the mouse 1005.

Further, the cross section setting unit 508 sets a cross section in thefirst three-dimensional image based on the acquired information. Then,the cross section setting unit 508 outputs the position and orientationof a plane representing the cross section in the first three-dimensionalimage to the cross-sectional image generation unit 510 and further tothe corresponding cross-sectional image generation unit 511.

The cross-sectional image generation unit 510 generates an image of thecross section having been set in the first three-dimensional image. Tothis end, the cross-sectional image generation unit 510 acquires thefirst three-dimensional image as an output of the three-dimensionalimage acquisition unit 504 and the position and orientation of the crosssection as an output of the cross section setting unit 508.

Then, the cross-sectional image generation unit 510 generates across-sectional image based on the acquired image and information andoutputs the generated image to an image synthesis unit 520.

The corresponding cross-sectional image generation unit 511 generates animage of a corresponding cross section in the first three-dimensionalimage, which corresponds to the cross section having been set in thefirst three-dimensional image, based on various information having beenacquired. Then, the corresponding cross-sectional image generation unit511 outputs the generated image of the corresponding cross section tothe image synthesis unit 520.

The processing to be performed by the corresponding cross-sectionalimage generation unit 511 is described below in more detail withreference to the flowchart, which illustrates an example procedure ofthe entire processing that can be performed by the image processingapparatus 500.

The image synthesis unit 520 combines the image of the cross sectionwith the image of the corresponding cross section and displays thecombined image. To this end, the image synthesis unit 520 acquires theimage of the cross section as an output of the cross-sectional imagegeneration unit 510 and the image of the corresponding cross section asan output of the corresponding cross-sectional image generation unit511. Then, the image synthesis unit 520 combines the acquired images anddisplays a combined image.

A basic configuration of a computer that executes a computer program (asoftware program) to realize the above-described functions of respectivecomponents constituting the image processing apparatus 500 is similar tothat of the computer illustrated in FIG. 2 according to the firstexemplary embodiment.

FIG. 6 is a flowchart illustrating an example procedure of the entireprocessing that can be implemented by the image processing apparatus500. In the present exemplary embodiment, the CPU 1001 executes theprogram that can realize the functions of respective constituentcomponents so as to reflect the flowchart illustrated in FIG. 6.

In the present exemplary embodiment, it is assumed that the programcodes relating to the flowchart illustrated in FIG. 6 are alreadyloaded, for example, from the external storage device 1007 to the RAM1002, before starting the following processing.

In step S6000, the image processing apparatus 500 executes, as exampleprocessing that can be implemented by the three-dimensional imageacquisition unit 504, processing for acquiring the first and secondthree-dimensional images and the three-dimensional displacement vectorgroup that are input to the image processing apparatus 100.

In step S6005, the image processing apparatus 500 executes, as exampleprocessing that can be implemented by the position acquisition unit 503,processing for acquiring the first lesion position data (e.g., acentroid position of a lesion illustrated in FIG. 7) that is input fromthe data server 590 to the image processing apparatus 100.

In step S6010, the image processing apparatus 500 executes, as exampleprocessing that can be implemented by the deformation data generationunit 506, processing for calculating the second lesion position (i.e.,the position in the second three-dimensional image that corresponds tothe first lesion position) based on the acquired three-dimensionaldisplacement vector group.

For example, the image processing apparatus 500 selects a voxelconstituting the first three-dimensional image that is positioned mostclosely to the first lesion position. Then, the image processingapparatus 500 obtains the second lesion position by adding athree-dimensional displacement vector at the selected voxel position tothe first lesion position. According to the example illustrated in FIG.7, the image processing apparatus 500 calculates the centroid position(see 710) of the lesion.

In step S6030, the image processing apparatus 500 executes, as exampleprocessing that can be implemented by the cross section setting unit508, processing for setting the position and orientation of a plane thatrepresents a cross section based on a designation by an operator.

In step S6040, the image processing apparatus 500 executes, as exampleprocessing that can be implemented by the cross-sectional imagegeneration unit 510, processing for generating a cross-sectional imagethat can be obtained by segmenting a predetermined range of the crosssection having been set in step S6030 from the first three-dimensionalimage acquired in step S6000.

Next, the image processing apparatus 500 executes, as example processingthat can be implemented by an approximate plane calculation unit (notillustrated) provided in the corresponding cross-sectional imagegeneration unit 511, processing for calculating an approximate planethat is a plane approximating a corresponding cross section in thesecond three-dimensional image through the following processing of stepS6050 to step S6070.

The processing to be performed in step S6050 is similar to theprocessing performed in step S3050 described in the first exemplaryembodiment, although it is not described below in detail.

In step S6060, the image processing apparatus 500 executes, as exampleprocessing that can be implemented by the corresponding cross-sectionalimage generation unit 511, processing for displacing the positions ofrespective grid points calculated in step S6050 based on thethree-dimensional displacement vector group acquired in step S6000.

Then, the image processing apparatus 500 calculates the positions ofpoint group (corresponding point group) in the second three-dimensionalimage that correspond to the displaced positions of the grid point groupin the first three-dimensional image. More specifically, the imageprocessing apparatus 500 selects, for example, a voxel constituting thefirst three-dimensional image that is positioned most closely to theposition x_(sn) (n=1 to N) of each grid point.

Then, the image processing apparatus 500 calculates the position x_(dn)(n=1 to N) of each corresponding point in the second three-dimensionalimage by adding a three-dimensional displacement vector at the selectedvoxel position to the positions of respective grid points.

In step S6070, the image processing apparatus 500 executes, as exampleprocessing that can be implemented by the corresponding cross-sectionalimage generation unit 511, processing described below. Morespecifically, the image processing apparatus 500 calculates anapproximate plane of the corresponding point group based on the firstlesion position acquired in step S6000, the second lesion positioncalculated in step S6010, and the positions of the point group(corresponding point group) in the second three-dimensional imagecalculated in step S6060.

More specifically, the image processing apparatus 500 calculates anapproximate plane that fits to the corresponding point group x_(dn) (n=1to N) using the least squares method on condition that the distance fromthe first lesion position to the cross section is equal to the distancefrom the second lesion position to the approximate plane.

Further, if an operator designates a desired approximate planecalculation method using the UI (not illustrated), the correspondingcross-sectional image generation unit 111 calculates an approximateplane according to the designated calculation method. For example, thecorresponding cross-sectional image generation unit 111 can employ acalculation method including weighting each corresponding pointaccording to the distance from the second lesion position to eachcorresponding point and obtaining an approximate plane according to theleast squares method reflecting the weighting.

Further, instead of using the information relating to the lesionposition, the corresponding cross-sectional image generation unit 111can employ a calculation method similar to the processing performed instep S3070 described in the first exemplary embodiment, such as acalculation method for obtaining a plane that can minimize a sum ofdistances from corresponding point group according to the least squaresmethod.

The processing to be performed after the above-described calculation ofthe plane (i.e., estimation of the in-plane moving component andcalculation of a segmenting range of the corresponding cross-sectionalimage) is similar to the processing performed in step S3070 described inthe first exemplary embodiment although its description is not repeated.

In step S6080, the image processing apparatus 500 executes, as exampleprocessing that can be implemented by the corresponding cross-sectionalimage generation unit 511, processing for generating a correspondingcross-sectional image by segmenting the predetermined range of theapproximate plane calculated in step S6070 from the firstthree-dimensional image.

In step S6090, the image processing apparatus 500 executes, as exampleprocessing that can be implemented by the image synthesis unit 520,processing for displaying the cross-sectional image generated in stepS6040 and the corresponding cross-sectional image generated in stepS6080 on the display unit 1006.

More specifically, as illustrated in FIG. 7, the image processingapparatus 500 can display a prone position cross-sectional image 702(i.e., a cross-sectional image) in such a manner that a nipple 703substantially faces on the lower side of a screen. Further, the imageprocessing apparatus 500 can display a supine position cross-sectionalimage 700 (i.e., a corresponding cross-sectional image) in such a mannerthat a nipple 701 substantially faces upward on the upper side of thescreen.

In this case, the image processing apparatus 500 determines the layoutrealizing a mirror image relationship between two images 700 and 702,according to which the images 700 and 702 are substantially symmetricalin the vertical direction. To this end, the image processing apparatus500 reverses the prone position cross-sectional image in the right andleft direction.

Further, if an operator designates a desired cross-sectional imagedisplay mode using the UI (not illustrated), the image synthesis unit520 displays a cross-sectional image according to the designated displaymode.

For example, the image processing apparatus 500 can employ a method fordisplaying a nipple facing upward in each of the prone positioncross-sectional image and the supine position cross-sectional imagewithout reversing the image in the right and left direction.

Further, the image processing apparatus 500 can employ a method fordisplaying the supine position cross-sectional image including a nipplefacing upward in a reversed state in the right and left direction anddisplaying the prone position cross-sectional image including a nipplefacing downward in a non-reversed state in the right and left direction(more specifically, the image to be reversed in the right and leftdirection is switched from the prone position to the supine position).

In step S6100, the image processing apparatus 500 determines whether theentire processing has been completed. For example, an operator can clickthe end button disposed on the display unit 1006 with the mouse 1005 toinput a termination instruction to the image processing apparatus 500.If it is determined that the entire processing has been completed (YESin step S6100), the image processing apparatus 500 terminates theprocessing of the flowchart illustrated in FIG. 6.

On the other hand, if it is determined that the entire processing hasnot been completed (NO in step S6100), the processing returns to stepS6030 to repeat the above-described processing in step S6030 to stepS6090.

As described above, the image processing apparatus 500 can executeprocessing in a manner capable of realizing the present invention.

As described above, the image processing apparatus according to thepresent exemplary embodiment calculates a plane that approximates acorresponding cross section of a deformed three-dimensional imagecorresponding to a cross section of a non-deformed three-dimensionalimage based on the position of a lesion.

Therefore, the image processing apparatus according to the presentexemplary embodiment can calculate a corresponding cross section withoutdeforming two three-dimensional images into the same shape, and candisplay these images

Further, the image processing apparatus according to the presentexemplary embodiment can realize a correlated display enabling users toeasily compare a non-deformed cross-sectional image and a deformedcross-sectional image when a deformation occurs due to a change in theposture of a diagnosis target.

In the above-described exemplary embodiment, the image processingapparatus designates a cross section in a non-deformed three-dimensionalimage and generates a corresponding cross-sectional image thatcorresponds to a deformed three-dimensional image.

However, similar to the first exemplary embodiment, the image processingapparatus can designate a cross section in a deformed three-dimensionalimage and generate a corresponding cross-sectional image thatcorresponds to a non-deformed three-dimensional image.

A corresponding cross-sectional image generation method to be used inthis case is similar to the method described in the first exemplaryembodiment, although it is not described below. Further, in a case wheredesignation of a cross section is feasible in either a non-deformedthree-dimensional image or a deformed three-dimensional image, if across section in one three-dimensional image is designated, the imageprocessing apparatus can generate and display a corresponding crosssection of the other three-dimensional image.

In the above-described exemplary embodiment, the data server 590 storeseach first lesion position representing the lesion position in the firstthree-dimensional image. However, instead of causing the data server 590to store lesion position information, the image processing apparatus 500can perform setting of the first lesion position as part of theprocessing to be performed.

For example, when a general UI that enables an operator to point anarbitrary coordinate position in a three-dimensional image is available,the operator can designate a centroid position of a lesion, if found inthe first three-dimensional image, and set the designated position as afirst lesion position.

For example, when a lesion is included in a cross-sectional image (see alesion 712 illustrated in FIG. 7), the operator can designate a centroidof the lesion (e.g., a central region of the lesion 712) with the mouse1005 to set the designated position as a first lesion position.

In this case, in step S6070, the approximate plane calculation unitcalculates an approximate plane based on the positions of acorresponding point group (i.e., not relying on the lesion position),similar to the first exemplary embodiment, until the first lesionposition is designated by an operator. Further, if the first lesionposition is set by the operator, then the approximate plane calculationunit starts calculating an approximate plane based on the positiondesignated by the operator.

The lesion position can be set based on an automatic detection of thelesion in the image processing. In this case, considering thereliability of the automatic lesion detection, it is useful to change aweighting factor to be used in calculating an approximate plane in stepS6070. For example, when the reliability can be expressed using anumerical value ranging from 0 to 1, it is useful to multiply theweighting factor by a numerical value representing the reliability.

The present invention is not limited to the above-described exemplaryembodiments. For example, the present invention can be embodied as asystem, an apparatus, a method, a program, or a storage medium. Morespecifically, the present invention can be applied to a system includinga plurality of devices or can be applied to an apparatus including asingle device.

According to the present invention, a software program can be directlyor remotely supplied to a system or an apparatus. A computer in thesystem or the apparatus can execute the program code to realize thefunctions of the above-described exemplary embodiments.

In this case, the program supplied to the system or the apparatus is thecomputer program relating to the flowcharts illustrated in theabove-described exemplary embodiments.

Accordingly, the present invention encompasses the program code itselfinstalled on the computer to cause the computer to realize the functionsand processes according to the present invention. Namely, the presentinvention encompasses a computer program itself that can realize thefunctions and processes according to the present invention.

In this case, object codes, interpreter programs, and OS script data areusable if they possess functions equivalent to the above-describedcomputer program.

A computer-readable storage medium supplying the computer program can beselected from any one of a Floppy® disk, a hard disk, an optical disk, amagneto-optical (MO) disk, a compact disc-ROM (CD-ROM), a CD-recordable(CD-R), a CD-rewritable (CD-RW), a magnetic tape, a nonvolatile memorycard, a ROM, and a digital versatile disc (DVD (DVD-ROM, DVD-R)).

The method for supplying the program includes accessing a website on theInternet using the browsing function of a client computer, when thewebsite allows each user to download the computer program of the presentinvention, or compressed files of the programs having automaticinstalling functions, to a hard disk or other recording medium of theuser.

Furthermore, the program code constituting the program of the presentinvention is dividable into a plurality of files so that respectivefiles are downloadable from different websites. Namely, the presentinvention encompasses World Wide Web (WWW) servers that allow numeroususers to download the program files so that their computers can realizethe functions or processes according to the present invention.

Moreover, enciphering the program according to the present invention andstoring the enciphered program on a CD-ROM or a comparable storagemedium is an exemplary method when the program of the present inventionis distributed to users. The authorized users (i.e., users satisfyingpredetermined conditions) are allowed to download key information from awebsite on the Internet. The users can decipher the program with theobtained key information and can install the program on their computers.

When the computer reads and executes the installed program, the computercan realize the functions of the above-described exemplary embodiments.

Moreover, an operating system (OS) or other application software runningon a computer can execute part or all of actual processing based oninstructions of the programs to realize the functions of theabove-described exemplary embodiments.

Additionally, the program read out of a storage medium can be writteninto a memory of a function expansion board inserted in a computer orinto a memory of a function expansion unit connected to the computer. Inthis case, based on instructions of the program, a CPU provided on thefunction expansion board or the function expansion unit can execute partor all of the processing to realize the functions of the above-describedexemplary embodiments.

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all modifications, equivalent structures, and functions.

This application claims priority from Japanese Patent Application No.2009-288457 filed Dec. 18, 2009, which is hereby incorporated byreference herein in its entirety.

What is claimed is:
 1. An image processing apparatus comprising: anacquisition unit configured to acquire a tomographic image of the humanbody in a first body position and to acquire a first three-dimensionalimage of the human body in a second body position, the tomographic imagebeing captured by a first image capturing apparatus and the firstthree-dimensional image being captured by a second image capturingapparatus; a first generation unit configured to generate deformationdata indicating a deformation of the first three-dimensional image fromthe second body position to the first body position; a second generationunit configured to generate a first cross-sectional image of the firstthree-dimensional image, and a second cross-sectional image based on thedeformation data, wherein a position and a posture of the firstcross-sectional image, the second cross-sectional image and thetomographic image correspond to each other; and a control unitconfigured to control a display unit to display the firstcross-sectional image, the second cross-sectional image and thetomographic image, so that the first cross-sectional image and thesecond cross-sectional image are arranged in a mirror imagerelationship.
 2. The image processing apparatus according to claim 1,wherein the second generation unit is configured to obtain a plane thatapproximates a curved surface corresponding to a cross section in thefirst three-dimensional image based on the correlation information andis further configured to generate the first cross-sectional image basedon the obtained plane.
 3. The image processing apparatus according toclaim 1, further comprising: a position acquisition unit configured toacquire a position in the first three-dimensional image or in a secondthree-dimensional image generated by deforming the firstthree-dimensional image based on the generated deformation data, whereinthe second generation unit is configured to generate the firstcross-sectional image based on the correlation information and theacquired position.
 4. The image processing apparatus according to claim1, wherein the first three-dimensional image is an image obtained bycapturing an image of the target object and a second three-dimensionalimage is an image obtained by deforming the first three-dimensionalimage, and the image processing apparatus further comprises a crosssection setting unit configured to set a cross section of thetomographic image in the second three-dimensional image, wherein thesecond generation unit is configured to generate the firstcross-sectional image corresponding to the cross section having been setby the cross section setting unit.
 5. The image processing apparatusaccording to claim 1, wherein the first three-dimensional image is animage of a breast captured in a prone position, and the secondthree-dimensional image is an image of a breast held in a supineposition that is obtained by deforming the first three-dimensionalimage, and the image processing apparatus further comprises a controlunit configured to cause a display device to display the secondcross-sectional image in such a manner that a nipple substantially facesupward and the first cross-sectional image relating to the firstthree-dimensional image in such a manner that a nipple substantiallyfaces downward.
 6. The image processing apparatus according to claim 1,wherein the first three-dimensional image is an image captured with anMRI apparatus.
 7. An image processing method comprising: acquiring atomographic image of human body in a first body position and a firstthree-dimensional image of the human body in a second body position, thetomographic image being captured by a first image capturing apparatusand the first three-dimensional image being captured by a second imagecapturing apparatus; generating deformation data indicating adeformation of the first three-dimensional image from the second bodyposition to the first body position; generating a first cross-sectionalimage of the first three-dimensional image, and generating a secondcross-sectional image based on the deformation data, wherein a positionand a posture of the cross-sectional image, the second cross-sectionalimage and a tomographic image correspond to each other; and controllinga display unit to display the first cross-sectional image, the secondcross-sectional image and the tomographic image, so that the firstcross-sectional image and the second cross-sectional image are arrangedin a mirror image relationship.
 8. A non-transitory computer readablestorage medium for storing a computer program that causes a computer toimplement an image processing method, the computer program comprising:computer-executable instructions for acquiring a tomographic image of ahuman body in a first body position and a first three-dimensional imageof the human body in a second body position, the tomographic image beingcaptured by a second image capturing apparatus; Computer-executableinstruction for generating deformation data indicating deformation ofthe first three-dimensional image from the second body position to thefirst body position; Computer-executable instructions for generating afirst cross-sectional image of the first three-dimensional image, and asecond cross-sectional image based on the deformation data, whereinapposition and a posture of the first cross-sectional image, the secondcross-sectional image and the tomographic image corresponding to eachother; And computer-executable instructions for controlling a displayunit to display the first cross-sectional image, the secondcross-sectional image and tomographic image, so that the firstcross-sectional image and the second cross-sectional image are arrangedin a mirror image relationship.
 9. An image displaying apparatuscomprising: an acquisition unit configured to acquire a tomographicimage of a breast of a human body in a supine position and a acquire afirst three-dimensional image of the breast of the human body in a proneposition, the tomographic image being captured by an ultrasonic imageapparatus and the first three-dimensional image being captured by one ofa magnetic resonance imaging apparatus, and X-ray computed tomographyimaging apparatus, a photo acoustic tomography apparatus, an opticalcoherence tomography apparatus, a positron emission tomography apparatusand a single photon emission computed tomography apparatus; a firstgeneration unit configured to generate deformation data indicating adeformation of the first three-dimensional image from the prone positionto the supine position; a second generation unit configured to generatea first cross-sectional image of the first three-dimensional image, anda second cross-sectional image based on the deformation data, wherein aposition and a posture of the first cross-sectional image, the secondcross-sectional image and the tomographic image correspond to eachother; and a display control unit configured to display the firstcross-sectional image and the second cross-sectional image that arearranged in a minor image relationship, according to which a nipple ofthe breast facing upward is positioned on an upper region of a screenand the nipple facing downward is positioned on a lower region of thescreen.